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Background of the Invention 
Field of the Invention 

The present invention relates to a correction 
method of distortion on an input image. 

Description of the Related Art 

Lately, the improvement of a slip recognition 
technology using an overhead reader (OHR) has been a 
major problem in financial optical character reader 
(OCR) business. 

An overhead reader is a stand type image input 
device using a line or area CCD as a camera element 
as shown in Fig. 1A. Compared with the conventional 
contact type image input device, such as an image 
scanner, etc., an OHR has an advantage that by using 
an OHR, a user can fill in a slip while inputting an 
image or can input an image while viewing a table in 
a slip, which leads to comfortable work. 

However, compared with an image obtained by a 
scanner (hereinafter called a "scanner image") , a slip 
image obtained by an OHR (hereinafter called an "OHR 
image") has degradation in image quality, such as uneven 



gradation, shadow, image distortion, etc. The main 
distortion of an OHR image is three-dimension 
distortion . 

A plurality of pieces of three-dimension 
distortion of an OHR image are classified into the 
following three categories. 

a) Center-folded distortion: Distortion caused 
on a slip with a fold at the center. There 
are two types of center-folded distortion: 
sunken center-folded distortion and raised 
center-folded distortion, in which the fold 
is sunk and raised, respectively. Figs. IB 
and 1C show an example of an OHR image with 
sunken center- folded distortion and an 
example of an OHR image with raised 
center- folded distortion, respectively . 

b) Circumference distortion: Distortion caused 
on the circumference of a slip. There are two 
types of circumference distortion: sunken 
circumference distortion and raised 
circumference distortion in which the fold 
is sunk and raised, respectively. Figs. ID 
and IE show an example of an OHR image with 
sunk circumference distortion and an example 
of an OHR image with raised circumference 



distortion, respectively . 

c) Pamphlet distortion: Distortion caused in the 
case of a pamphlet. Fig* IF shows an example 
of pamphlet distortion. 

In order to use an OHR in financial OCR business, 
a correction method for overcoming three-dimension 
distortion in these images must be developed, which 
is a major problem. 

Japanese Patent Laid-open No. 10-150532 
discloses a distortion correction method for an OHR 
image with three-dimension distortion in order to 
implement high-definition document recognition. This 
correction method targets one-point stapled 
distortion caused on a document obtained by stapling 
several pieces of paper at one point. A plurality of 
pieces of images photographed from a plurality of top 
and side viewpoints in order to calculate a 
three-dimension shape are used to measure 
three-dimension distortion* 

However, since the conventional method for 
correcting three-dimension distortion uses a 
plurality of cameras, implementation cost becomes high 
and wide installation space is required. 



4 



Summary of the Invention 

It is an obj ect of the present invention to reduce 
both implementation cost and installation space by 
using only one piece of image photographed from the 
5 top . 

In the present invention, a three-dimension 
curved-surface model is estimated by using 
two-dimension outlines obtained from a piece of image 
photographed from the top as a clue and further using 

10 a restriction that a piece of paper is rectangular, 
and three-dimension distortion in the image is 
corrected using the calculated three-dimension 
curved-surface model . 

According to the present invention, an outline 

15 extraction device comprises a curved-surface 
estimation device and a distortion correction device • 
An image obtained by photographing a piece of 
rectangular paper is used an input image. The outline 
extraction device extracts the outline of the paper 

20 from the input image, and the curved- surface estimation 
device estimates the three-dimension curved-surface 
model of the paper using the two-dimension outline as 
a clue. Then, the distortion correction device corrects 
the distortion based on the estimated three-dimension 

25 curved-surface model and outputs the 
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distortion-corrected image. 

Since by adopting such a configuration, an input 
image is corrected using two-dimension outline 
distortion obtained from a piece of Image as a clue, 
5 compared with the conventional technology, one camera 
is sufficient and as a result, both implement cost and 
installation space can be reduced. 

Brief Descriptions of the Drawings 

10 Fig. 1A shows the appearance of an OHR. 

Fig . IB shows an example of a sunken center-folded 
OHR image . 

Fig. 1C shows an example of a sunken center- folded 
OHR image. 

15 Fig. ID shows an example of a (raised) 

circumstance-distortion OHR image. 

Fig. IE shows an example of a (sunken) 
circumstance-distortion OHR image. 

Fig. IF shows an example of a pamphlet-distortion 
2 0 OHR image. 

Fig. 2A shows the basic configuration of the 
present invention . 

Fig. 2B shows the configuration of the first 
preferred embodiment of the present invention. 
25 Fig. 3 shows how to evaluate outline likeliness 
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in the outline extraction device. 

Fig. 4 shows outline lines, outline vertexes and 
sample outline points. 

Fig. 5 shows the location relationship among a 
5 camera center, a two-dimension outline and a 
three-dimension outline in a (upper) sunken 
center-folded curved-surface model. 

Fig. 6 shows a linear Coons curved-surface. 

Fig. 7 shows a curved coordinate system on a 
10 three-dimension curved-surface. 

Fig. 8 shows the correction result of sunken 
center-folded distortion. 

Fig. 9 shows the configuration of the second 
preferred embodiment of the present invention. 
15 Fig. 10 shows outline lines, outline vertexes 

and sample outline points. 

Fig. 11 shows the location relationship among 
a camera center, a two-dimension outline and a 
three-dimension outline in a (upper) raised 
20 center-folded curved-surface model. 

Fig. 12 shows the (entire) raised center-folded 
curved-surface model . 

Fig. 13 shows the correction result of raised 
center- folded distortion . 
25 Fig. 14 shows the configuration in the case where 
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distortion is corrected without estimating a curved 
surface . 

Fig. 15 shows the system adopting the 
configuration shown in Fig. 14. 
5 Fig. 16 shows the basic configuration of an 

information processing device. 

Fig. 17 shows how a software program, etc., is 
provided. 

10 Descriptions of Preferred Embodiments 

The preferred embodiments of the present 
invention are described with reference to the drawings . 

Fig. 2A shows the basic configuration of the 
present invention. The apparatus comprises an outline 

15 extraction device 101, a curved-surface estimation 
device 102 and a distortion correction device 103 . When 
an image obtained by photographing a piece of 
rectangular paper is inputted, the outline extraction 
device 101 extracts the outline of the paper from the 

20 input image, and the curved-surface estimation device 
102 estimates the three-dimension curved-surface 
model of the paper using two -dimension outlines as clues . 
Then, the distortion correction device 103 corrects 
the distortion based on the estimated three-dimension 

25 curved-surface model 'and outputs the 
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distortion-corrected image . 

Although a shadow is caused due to a fold, etc., 
on the OHR image of a slip, the acquisition of stable 
information about the shadow cannot be expected due 
5 to the dynamic shadow and uneven gradation of the shadow. 
Therefore, it is not recommended to use the fluctuation 
of a shadow as a clue to distortion correction. 
Therefore, in the present invention shown in Fig. 2A, 
distortion is corrected using as a clue a slip outline 

10 that can be relatively stably obtained. In this method, 
attention is focussed on the fact that a slip is 
rectangular, and an image is horizontally/vertically 
enlarged/reduced using a transformed slip outline as 
a clue. The method can be said one type of "shape from 

15 contour" methods. The feature of this method is that 
the curved-surface estimation device 102 estimates a 
three-dimension curved-surface model from a piece of 
distorted image using two-dimension outline 
distortion. Since one camera is sufficient, 

20 implementation cost can be greatly reduced. 
Furthermore, since both shadow fluctuation and slip 
outline distortion that can be relatively stably 
extracted are used as clues, high-definition 
distortion correction can be implemented. 

25 Fig. 2B shows the configuration of the first 
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preferred embodiment of the present invention. The 
distortion correction method shown in Fig. 2B corrects 
the sunken center-folded distortion of a plurality of 
pieces of distortion caused on a slip with a fold at 
5 the center. As shown in Fig. IB, in a sunken 
center-folded OHR image, the top outline, bottom 
outline and a center-folded outline can be recognized 
as line segments touching a reference plane, such as 
a table, etc* Sometimes the left outline, right outline 

10 and center-folded outline are recognized as line 
segments depending on how to place a slip. However, 
in such a case, if the image is rotated 90 degrees, 
the same situation as described above can be obtained. 
Therefore, the first preferred embodiment that 

15 corrects sunken center-folded distortion is described 
in detail below using as an example the case where the 
top outline, center-folded outline and bottom outline 
can be recognized as line segments. 

In Fig. 2B, when an image obtained by 

20 photographing a piece of rectangular paper is inputted, 
an outline extraction device 201 extracts the outline 
of the paper from the input image, and a sunken 
center-folded curved-surface estimation device 202 
estimates a three -dimension curved- surface model 

25 using the outline distortion of a sunken center-folded 
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slip as a clue. Then, a distortion correction device 
203 corrects the distortion based on the estimated 
three-dimension curved-surface model of the paper and 
outputs the distortion-corrected image. 
5 The outline extraction device 201 first 

designates outlines obtained by edge extraction, such 
as Sobel's edge extraction, etc., as outline pixel 
candidates and judges whether each outline pixel 
candidate constitutes the outline. The outline 

10 extraction device 201 can also be designed to extract 
and to use not only a paper outline obtained by edge 
extraction, but also a straight line 
(horizontal/vertical), a character string that are 
provided in a piece of paper. 

15 A case where it is judged whether a target pixel 

in an aggregate of extracted outline pixel candidates 
constitutes, for example, the left outline (see Fig. 
3) . If a specific pixel constitutes the left outline, 
the left side of the pixel is a pixel string with a 

20 low pixel value, such as a background color and the 
right side is a pixel string with a high pixel value, 
such as white. Therefore, for example, if the product 
sum of a linear filter 302 (-1, -1, -1, 0, 1, 1, 1) 
and an outline pixel 301 is calculated, the product 

25 sum value becomes large. Therefore, it is all right 
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if a specific threshold value is pre-determined and 
it is judged that an outline pixel candidate with a 
product sum larger than the threshold value is an 
outline pixel. In this way, the left outline is 
5 determined by calculating the product sum of the linear 
filter 302 and each outline pixel candidate and by 
comparing the obtained product sum with the threshold 
value . 

Similarly, in the case of the right outline, it 
10 is all right if a linear filter (1, 1, 1, 0, -1, -1, 
-1) with a configuration the reversal of the left 
outline is used. The top/bottom outline can also be 
determined in the same way. 

A linear filter can have arbitrary peculiar 
15 vector that is horizontally/vertically symmetric with 
0 at the center. Specifically, the filter can have 
peculiar vector in which n arbitrary constants k and 
-k arrayed with 0 at the center (k, k, k, 0, -k, 

k, k ) • 

20 As described above, the outline extraction 

device 201 outputs a binary outline image . Furthermore, 
the outline extraction device 201 also extracts outline 
vertexes and sample outline points. Since an outline 
line, an outline vertex and an sample outline point 

25 are all related to an outline in a two-dimension image, 



it is hereinafter called as a two-dimension outline 
line, a two-dimension outline vertex and a 
two-dimension sample outline point, respectively, by 
attaching "two-dimension" to the respective heads in 
order to distinguish them from slip outlines in a 
three-dimension image. Fig. 4 shows expressions based 
on a two-dimension coordinates of an two-dimension 
outline line, an two-dimension outline vertex and an 
two-dimension sample outline point that are obtained 
by the outline extraction device 201. 

A two-dimension outline vertex can be obtained 
by horizontally following an outline line from an 
outline point around the center of the top outline AB 
or bottom outline EF shown in Fig. 4 and searching for 
outline points, the inclination of which suddenly 
change (top left outline vertex A, top right outline 
vertex B, lower left outline vertex E and lower right 
outline vertex F) . The left/right vertexes of a 
center-folded outline (slip center) are extracted 
according to the location relationship with a rectangle 
ABFE that is obtained by connecting outline vertexes 
A, B, E and F by line segments. In the case of a sunken 
center-folded slip, left/right outline points that 
deepest enter into the rectangle are designated as 
sunken center-folded outline vertexes (broken-line 



left vertex C and broken-line right vertex D) . 

Furthermore, a two-dimension sample outline 
point used to apply segment linear approximation to 
extracted left/right outline lines can be obtained by 
selecting them at predetermined equal intervals 
towards the Y axis shown in Fig. 4. 

In the following description, the same 
expressions of eachpoint, line segment, outline lines , 
etc., based on a two-dimension coordinates as those 
in Fig . 4 are used . 

Next, the sunken center- folded curved-surface 
estimation device 2 02 is described. The sunk 
center-folded curved-surface estimation device 202 
calculates the coordinates of the three-dimension 
sample outline point of a sunk center-folded slip 
assuming that the point is an intersection point of 
a line segment connecting obtained two-dimension 
sample outline point and a camera center, which is the 
viewpoint of an input image, (since a stand type OHR 
is used, a camera center is predetermined) and a plane 
that passes through a line connecting two two-dimension 
outline vertexes and that is perpendicular to the 
reference plane. The device 202 designates a broken 
line obtained by connecting the calculated 
three-dimension sample outline points as an outline 
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line and estimates a curved-surface obtained by 
applying linear interpolation approximation to the 
outline line, as the three-dimension curved-surface 
model of a distorted slip curved-surface. 
5 A sunken center-folded curved-surface can be 

horizontally divided into two curved surfaces with a 
center-folded outline (line segment touching the 
reference plane of the slip center) as a boundary. The 
process of the sunken center-folded curved-surface 
10 estimation device 202 is described in detail below 
limiting the process to the upper curved-surface. As 
for the lower curved-surface, the same description 
applies . 

Fig. 5 shows the location relationship among a 
15 camera center, a two-dimension outline and a 
three-dimension outline . The meanings of symbols shown 
in Fig. 5 are as follows. 

x : Three-dimension vector 

20 X : Two-dimension vector 

x 7 y,z : Three-dimension coordinate value 
X,Y : Two-dimension coordinate value 
L : Left 
R : Right 

25 /: Suffix of a left outline point, i = 0 y -~,m 
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j: Suffix of a right outline point, j-0,*- y n 

A point on a three-dimension outline that is 
related to a two-dimension sample outline point by 
5 perspective conversion is called a three-dimension 
sample outline point. The coordinates of both a camera 
center and a three-dimension sample outline point are 
assumed to be expressed as shown in Fig. 5* The 
coordinates of a three-dimension sample outline point 
10 on the left outline line are calculated as follows. 



three-dimension left outline line corresponding to a 
two-dimension sample outline point on the 
two-dimension left outline line can be calculated as 

15 an intersection point of a line segment connecting a 
camera center and the two-dimension sample outline 
point, and a plane that pass through a line segment 
AC connecting left outline vertexes A and C and that 
is perpendicular to xy plane. 

20 The equation (1) of a plane perpendicular to xy 

plane, including line segment AC can be expressed as 
follows . 



A three-dimension sample outline point on the 




) and C: (X, 




(1) 
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If it is assumed that a three-dimension sample 
outline point is a point where a line segment connecting 
the camera center and two-dimension sample outline 
point is internally divided at a ratio of (1-t) : t, 
5 the following equation (2) holds true. 

xf = ATf +(x K -X?)t 
<y> =l L +(y K -Y^t (2) 

However, in the above equation, 
Three-dimension sample outline point: xf = {x^ ,yf 
Camera center K: (x K ,y K 9 z K ) 
10 Two-dimension sample outline point: Xf = (X^ 9 Y t L ) . 

If t is solved from equation (1) and (2), the 
following equation (3) is obtained. 

t _ (x L m - xf- xff - ) - off - y, l )(x£ - xj ) 

(x K -X^){Y 0 L -Y')-(y K -Yfy&i -X L J 

If t obtained according to equation (3) is 
15 assigned to equation (1) and (2) , the coordinates of 
each three-dimension sample outline point on the left 
outline line can be calculated. 

The coordinates of a three-dimension sample 
outline point on the right outline can also be 
20 calculated in the same way. 

Broken lines obtained by connecting the 
three-dimension sample outline points with a line 
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segment are designated as the left /right outline lines , 
and a curved surface obtained by applying linear 
interpolation approximation to these outlines is 
designated as the three-dimension curved-surface 
5 model of a distorted slip. 

For the linear interpolation approximation, for 
example, an interpolation method, such as linear Coons 
interpolation, etc., can be used. Linear Coons 
interpolation means to interpolate space between four 

10 three-dimension outlines with line segments if there 
is assumed to be four three- dimension outlines . A linear 
Coons curved-surface, to which linear Coons 
interpolation is applied, is a kind of parametric 
spatial curved-surface that is defined by a curved 

15 coordinate system (u, w) . If it is assumed that a curved 
surface is surrounded by four curved surfaces P (u, 0) , 
P (u, 1), P(0, w) and P (1, w) , the coordinates Q (u, w) 
of a point on the curved surface can be expressed by 
the following equation (4) using an outline curve 

20 expression (see Fig. 6). A curved surface obtained 
according to equation (4) is designated as a 
three-dimension curved-surface model. 

Q(u, w) = P(//,0)(1 - w) + P{u,\)w + P(0 7 w)(l - m) + P{\ w)u 

- P(0 ? 0)(1 - w)(l - w) - P(0,1)(1 -u)w ( 4 ) 

- P(1,0>(1 - w) - P{\\)uw 
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Although as a method for calculating a 
three-dimension outline model in the sunken 
center-folded curved-surface estimation device 202, 
a method in which points scattered on a three-dimension 
5 outline line are designated as three-dimension sample 
outline points and the height or corresponding value 
of each three-dimension sample point is used as a model 
parameter, is described, a method for calculating a 
three-dimension outline model is not limited to this. 
10 Specifically, for example, for the three-dimension 
outline model, a curved model with a parameter, such 
as a spline curve, a Bezier curve, etc., can also be 
used. 

A spline curve and a Bezier curve are parametric 
15 spatial curves. Generally, a parametric spatial curve 
is expressed by the following equation (5) . 

x - x(t) 

<y = y(t) (5) 

However, in the above equation, t is a parameter 
and (x, y, z) is the coordinates of a point on a 
20 three-dimension spatial curve. 

According to equation (5) , a spline curve is expressed 
by the following expression (6). 
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y{t) = ±B!r' 



(6) 



Similarly, according to equation (5), a Bezier 
curve is expressed by the following equation (7) . 

x( t )=f,B*j:(t) 

r=l 



(7; 



However, in the above equation, 



J" it) 



J i! (*-/)! 

Since both a spline curve and a Bezier curve can 
be expressed by parameter t , a three-dimension outline 
10 model can be calculated using this parameter t as a 
model parameter. 

Furthermore, a center-folded curved-surface 
estimation device (or curved-surface estimation 
device) can be designed to estimate a three-dimension 
15 curved surface in the same way by using information 
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about a straight line (vertical/horizontal) , a 
character string (vertical/horizontal) that are 
provided in a piece of paper, instead of outline lines . 

Next, the distortion correction device 203 is 
described. The distortion correction device 203 
obtains an image after correction, with the length of 
top/bottom outlines and the length of left/right 
outlines as width and height, respectively, by 
calculating a location in an input image corresponding 
to eachpixel of an image after correction using a curved 
coordinate system with an outline line as each 
coordinate axis, and by setting the value (binary, 
gradation or color) of a corresponding pixel in the 
input image as the target pixel value of the image after 
correction. 

In the case of a center-folded slip, both the 
top/bottom outline lines are obtained as line segments 
connecting outline vertexes. Both the left/right 
outline lines are obtained as broken lines connecting 
three-dimension sample outline points obtained by the 
sunken center- folded curved- surface estimation device 
202. A center-folded outline is obtained as a line 
segment connecting three-dimension sample outline 
points obtained by the sunken center-folded 
curved-surface estimation device 202. 
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The center-folded slip is horizontally divided 
and distortion is corrected by setting a curved 
coordinate system on each corresponding 
three-dimension curved surface. A distortion 
5 correctionmethod for the upper three-dimension curved 
surface is described below. The same process can also 
be applied to the lower curved surface. 

The top and left outline lines are assigned to 
X and Y axes, respectively, using the top left outline 

10 vertex (point A shown in Fig. 4} as an origin. For the 
width W of an image after correction, the average value 
of the top outline line length and the center-folded 
outline line length is used, and for the and height 
H of the image, the average value of the left outline 

15 line length and the right outline line length is used. 
Specifically, the width W and height H are calculated 
according to the following equation (8) and (9), 
respectively. 

W = -(AB + cd) (8) 

20 H =^(Ac + Bd) (9) 

Three-dimension coordinates are calculated by 
relating point L(Y) on the left outline line to Y with 
an integer value between 0 and height H in order to 



graduate Y coordinates on the left outline line. The 
three-dimension coordinates of L (Y) are calculated by 
following a broken line connecting three-dimension 
sample outline points representing the left outline 
line from the top left outline vertex and proceeding 
by length (H/Ac) at one time. 

Similarly, three-dimension coordinates are 
obtained by relating R ( Y) on the right outline line 
to Y with an integer value between 0 and height H. The 
three-dimension coordinates of R (Y) are calculated by 
following a broken line connecting three-dimension 
sample outline points representing the right outline 
line from the top right outline vertex and proceeding 
by length (H/Bd) at one time. It can be considered that 
a curved coordinate system is expressed by both L(Y) 
and R(Y) (see Fig. 7) . 

Then, the gradation G(X, Y) (0<X<W, 0<Y<H) of 
a pixel corresponding to the two-dimension coordinates 
(X, Y) of an image after distortion correction are 
calculated by defining a curved coordinate system in 
this way. 

Apoint corresponding to corrected two-dimension 
coordinates (X, Y) is point P with two-dimension curved 
coordinates (X, Y) on a three-dimension curved surface . 
It is assumed that the three-dimension coordinates of 
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point P are expressed by the following expression ( 10) . 
x p = (x p ,y p ,z p ) (10) 

Point P is a point obtained by applying linear 
interpolation to both point L(Y) on the left outline 
5 line and R(Y) on the right outline line and it can be 
expressed as a point obtained by internally dividing 
line segment L(Y)R(Y) at a ratio of X/W: (1-X/W). 
Therefore, the three-dimension coordinates of point 
P can be calculated according to the following equation 
10 (11) . 

(x p , y\ z p ) = (1 - ^)L(Y) + ^R(Y) (11) 

Then, the two-dimension coordinates of an 
intersection point of a straight line that starts from 
the camera center and passes through point P, and xy 
15 reference plane is calculated. The coordinates can be 
calculated according to equation (12) depending on 
perspective conversion conditions. 

z -z z -z 

P is the intersection point of a straight line 
20 passing through both the camera center and point P, 
and xy reference plane. 

Therefore, the gradation G (X, Y) of an image after 
correction can be calculated according to the following 
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equation 
G(X,Y) = G( 
= G( 

However, in the above equation, G{X,Y) 
indicates the gradation value of a pixel with the 

5 two-dimension coordinates (X,Y) of an image before 
correction* 

[x] is Gauss symbol and indicates the largest integer 
not exceeding x. 

However, in equation (13), the two-dimension 

10 coordinates of the intersection point of a straight 
line that starts from the camera center and passes 
through point P, and xy reference plane are rounded 
by counting fractions of 5 and over as a unit and 
disregarding the rest since it is a real number. 

15 Fig. 8 shows the correction result of a sunken 

center-folded distorted OHR image processed according 
to the first preferred embodiment of the present 
invention described above . Fig. 8 shows the correction 
of the image shown in Fig. IB. In this way, according 

20 to the first preferred embodiment of the present 
invention, a sunken center-folded distorted OHR image 
can be successfully corrected. 
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Fig* 9 shows the configuration of the second 
preferred embodiment of the present invention. The 
distortion correction method shown in Fig. 9 is used 
to correct a raised center-folded distortion of a 
5 plurality of pieces of distortion caused on a center- 
folded slip. As shown in Fig. 1C, although in a raised 
center-folded distorted OHR image, the top/bottom 
outlines of a slip touch a reference plane, such as 
a table, the center-folded outline is raised beyond 

10 the reference plane as not in a sunk center-folded 
distortion targeted in the first preferred embodiment . 
Sometimes the left outline, right outline or 
center-folded outline is recognized as a line segment . 
In this case, if the image is rotated 90 degrees, the 

15 same description applies. Therefore, the second 
preferred embodiment for correcting a raised 
center-folded distortion correction is described in 
detail below using as an example the case where the 
top, center-folded or bottom outline line is recognized 

20 as a line segment. 

In Fig. 9, when an image obtainedby photographing 
a piece of rectangular paper is inputted, an outline 
extraction device 901 extracts the outline of the paper, 
and a raised center-folded curved-surface estimation 

25 device 902 estimates a three-dimension curved-surface 



model using the outline distortion of a raised 
center-folded slip as a clue. Then, a distortion 
correction device 903 corrects the distortion based 
on the three-dimension curved-surface model of the 
paper and outputs the distortion-corrected image. 

In almost the same way as the outline extraction 
device 201 described in the first preferred embodiment, 
the outline extraction device 901 calculates the 
product sum of an edge-extracted outline pixel 
candidate and a linear filter after edge extraction 
and determines an outline pixel based on the calculated 
product sum value. Two-dimension outlines, 
two-dimension outline vertexes and two-dimension 
sample outline points are also extracted. 

In this case, in order to extract the left/right 
vertexes of a center-folded outline (slip center) , the 
left/right outline points that are farthest away from 
the rectangle are extracted as raised center-folded 
outline vertexes as not in the case of a sunken 
center-folded slip . 

The outline extraction device 901 extracts 
two-dimension outline lines, two-dimension sample 
outline points and two-dimension outline vertexes 
based on the two-dimension coordinates shown in Fig. 
10. In the following description, the same expressions 
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of each point/ a line segment, an outline line, etc., 
based on two-dimension coordinates as shown in Fig, 
10 are used. 

Next, the raised center- folded curved-surface 
5 estimation device 902 is described. In the raised 
center-folded curved-surface estimation device 902, 
the value equivalent to the height of the center-folded 
outline vertex at the center is designated as the 
two-dimension model parameter of a raised 

10 center-folded slip with the top/bottom outlines, the 
height from a reference plane is zero and the 
center- folded outline at the center, the height of which 
is unknown. Then, an energy function with a restriction 
that the length of the top three-dimension outline is 

15 the same as length of lower three-dimension outline 
and that the length of the left three-dimension outline 
is the same as the length of right three-dimension 
outline, is expressed using the model parameter. In 
order to minimize and optimize this energy function, 

20 an optimal model parameter is calculated by a repetition 
method, etc. A three-dimension curved-surface model 
is estimated by calculating the coordinates of each 
three-dimension sample outline point using the 
calculated model parameter. A method for calculating 

25 the model parameter is described below. 
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Since a raised center-folded curved surface can 
be horizontally divided with the center-folded outline 
(slip center) as a boundary, the process of the raised 
center- folded curved-surface estimation device 902 is 
5 described in detail using the upper curved surface as 
an example. 

Fig, 11 shows the location relationship among 
the camera center, a two-dimension outline and a 
three-dimension outline. The meaning of each symbol 

10 used in Fig. 11 is the same as that of External character 
line 1. A point on a three-dimension outline line that 
is related to a two-dimension sample outline point by 
perspective conversion is called a three-dimension 
sample outline point. Fig, 11 shows the coordinates 

15 of both the camera center and three-dimension sample 
outline point. 

If the followings are assumed, 
three-dimension sample outline point: xf = (xf 9 yf ,zf) 
camera center K: (x K y y K \z K ) 

20 two-dimension sample outline line: Xf -(X^Y^) 

and if three-dimension sample outline point xf 
internally divides a line segment K Xf connecting 
camera center K and two-dimension sample outline point 
at a ratio of (1-5,): 5, , the following equation (14) 

25 can be obtained. 
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<y"=Y, L +(y K -Y, L )s, (14) 
zf=z K s, 

If the followings are assumed, 
three-dimension sample outline point: =(x* ,z*) 
camera center K: (x K ,y K ,z K ) 
5 two-dimension sample outline line: X*=(X*,y*) 
and if three-dimension sample outline point 
internally divides a line segment K X* connecting 
camera center K and two-dimension sample outline point 
at a ratio of (1-*.):* , the following equation (15) 
10 can be obtained. 

xf=X« + ( X K -X«) tj 
^y^=Yf+{y K -Yf)t j (15) 

The coordinates of the three-dimension left 
vertex c and right vertex d on a raised center-folded 
outline can be expressed by the following equation (16) 

15 and (17), respectively. If the internal division ratio 
of a line segment connecting these two points is 
determined, the coordinates of each three-dimension 
sample outline point can be calculated on conditions 
that each three-dimension sample outline point is on 

20 a line segment connecting camera center K and a 
two-dimension sample outline point, and is on a plane 
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that includes a line segment Ac or Bd and is 
perpendicular to xy plane. 



division ratio between three-dimension left vertex c 
and right vertex d is the two-dimension model parameter 
of the three-dimension model of a slip curved-surface. 

The corresponding internal division ratio of the 

10 left outline line can be calculated on the conditions 
for three-dimension sample outline line described 
above using amodel parameter according to the following 
equation (18) if the followings are assumed, 
the internal division ratio of a three-dimension sample 

15 outline point: s r 

model parameter (internal division ratio of a 
three-dimension left vertex) : s . 



(16) 



(17) 



5 



Therefore, it can be said that the internal 



s , = 



(18) 



However, 



in the above equation, 



values 



20 a^yP^Xii art d S t L are calculated as follows. 



A L =(Yo -Y!-)X L m -{X L 0 -Xf)Yt+{X L J t L -xX L ) 
Zi = (** ~K)(y K ~Y 1 L )-{x K -Xf){y K -7 m L ) 



31 



Similarly, the corresponding internal division 
ratio of the right outline line can be calculated on 
5 the conditions for a three-division sample outline 
point using amodel parameter according to the following 
equation (19) if the followings are assumed, 
the internal division ratio of a three-dimension sample 
outline point: tj 
10 model parameter (internal division ratio of a 
three-dimension left vertex) : t n . 



However, in the above equation, values 
a %flf>Z% and &f are calculated as follows. 

15 

«? =(* K -X*Wo R -Y J R ) 
/3«=(Y 0 R -Y?)X R -(X R 

3 R =(x*-X R )(Y 0 R -Y n R ) 
20 

The minimization and optimization of energy 
function E expressing a restriction that 
three-dimension top/bottom outline lines are the same 
in length, and three-dimension left/right outline 



-x*)Y n R +(X«Y?-xfY 0 R ) 

- { yK-Y?){X»-X») 
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lines are the same in length, can be formulated and 
an optimal model parameter can be calculated, for 
example, by a repetition method as described later. 

Energy function E is defined by the linear sum 
5 of the squared difference in length between top outline 
line AB and raised center-folded outline line cd, and 
the squared difference in length between left outline 
line Ac and right outline line Bd. The value of energy 
function E is uniquely determined by an internal 
10 division ratio, which is the model parameter of a slip 
curved-surface. Therefore, energy function E can be 
expressed by the following equation (20) . 
E(s m J n )^k } E } + k 2 E 2 =k l (AB-cd) 2 + k 2 (Ac-Bd) 2 (20) 
where *, and k 2 are constants (for example, 
15 k } - I and k 2 -\)* 

Since both the top outline line (line segment 
AB) and a raised center-folded outline line (line 
segment cd) are line segments, they can be calculated 
from the length of three-dimension vertex coordinates 
20 according to the following equation (21) and (22), 
respectively . 

AB = J(XZ-X«) 2 +(Y 0 L -Y 0 R y (21) 
cd = 4{xi-x«Y + (yt-y R n ? + rt-z R n y (22) 

The length of the left outline line Ac and that 
25 of the right outline line Bd are approximated by the 
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respective sumof the length of line segments connecting 
three-dimension sample outline points. Specifically, 
they can be calculated according to the following 
equation (23) and (24) , respectively. 

5 Ac=f,y M -xf -*y +o*i -y.f + (23) 
-^hSM. -*?> 2 -^) 2 +«i -^> 2 < 24 > 

Amodel parameter for minimizing energy function 
E is calculated by the steepest descent method. 
Specifically, an approximate initial value is set and 
10 a model parameter is repeatedly changed according to 
the following equation (25) . In this case, s is a minute 
positive number. 



r 
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(25) 



Equation (25) is developed using the definition 
15 of the energy function of equation (20) , the following 
equation (26) and (27) can be obtained. 



3£ = .^L + *^1 



ds_ 



ds. 



(26) 



&_ dt_ 2 a*. 



(27) 
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However, each term is as follows. 

= 2{cd -ab)~ k - x* r - O + o£ - y" )(y K - > + (*« - )z * 1 



as. 



-2(«/ - iifl) • • [(x L m - x R n )(x K -x L J + (y L m - y* )(y K -y L m ) + ^ L m ~ £ V 
cd 



f 



10 
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= "f*r-if*r a=0 „_!), (_,-=„) 

In this way, a model parameter can be calculated 
and as a result, the coordinates of each three-dimension 
sample outline point canbe calculated. Then, the raised 
center-folded curved-surface estimation device 902 
designates broken lines connecting obtained 
three-dimension sample outline points as outline lines, 
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and estimates a curved-surface obtained by the linear 
interpolation approximation of the outlines. For the 
linear interpolation approximation, an interpolation 
method, such as a linear Coons interpolation, etc., 
5 is used as in the case of a sunken center-folded 
curved-surface estimation device 202. 

Although as energy function E, the energy 
function E of an upper raised center-folded 
curved-surface is formulated, the energy function E 
10 of the entire curved-surface can also be formulated. 
Energy function E obtained by formulating the entire 
curved-surface can be expressed by the following 
equation (28) as shown in Fig. 12. 

= k } (AB ~cdf + k 2 {Ac - Bdf + * 3 (EF - cdf + k 4 (cE - dFf 
15 (28) 
However, k ly k 2J k 3 ,k 4 are constants. 
In this case, since the optimization problem of 
the entire curved-surface can be solved, accuracy is 
further improved than in the case where a curved-surface 
20 is horizontally divided, and a model parameter is 
calculated for each divided curved-surface. 

Next, the distortion correction device 903 in 
the second preferred embodiment is the same as the 
distortion correction device 2 03 in the first preferred 
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embodiment. Specifically, the distortion correction 
device 903 obtains an image after correction with the 
length of the top/bottom outline and that of the 
left/right outline as width and height, respectively, 
5 by calculating the location in an input image, 
corresponding to eachpixel of an image after correction 
using a curved coordinate system with outline lines 
as coordinate axes and by setting the value (binary, 
gradation or color) of the corresponding pixel in the 

10 input image as the target pixel value of the image after 
correction. For detail, see the description of the first 
preferred embodiment . 

Fig, 13 shows the correction result of a raised 
center-folded distortion OHR image processed 

15 according to the second preferred embodiment of the 
present invention. In Fig. 13, the image shown in Fig. 
1C is corrected. In this way, according to the second 
preferred embodiment of the present invention, a raised 
center-folded distortion OHR image can be 

20 satisfactorily corrected. 

Although sunken/ raised center-folded 

distortion correction methods for a slip OHR image has 
been so far described in detail, the application is 
not limited to the sunken/raised center-folded 

25 distortion correction. Specifically, the same method 
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is also applicable to line-textured curved-surface 
distortion. Line-textured curved-surface is a general 
curved-surface caused when a piece of paper is 
distorted. 

5 For example, if there is raised circumference 

distortion (Fig. ID) on a slip OHR image as an example 
of line-textured curved-surface, all three-dimension 
outlines are curves. Therefore, although in the case 
of raised center-folded distortion, two model 

10 parameters are calculated (the second preferred 
embodiment of the present invention) , a greater number 
of model parameters (for example, 10 to 20) can also 
be set, an energy function representing a restriction 
that the top/bottom outlines are the same in length 

15 and the left/right outlines are the same in length can 
also be formulated and the optimization problem can 
also be solved. Then, a three-dimension curved-surf ace 
model can also be estimated from a calculated model 
parameter, distortion on an input image can also be 

20 corrected based on the estimated curved-surface model 
and the corrected image can also be output ted. Similarly, 
the three-dimension curved-surface models of both 
sunken circumference distortion (Fig. IE) andpamphlet 
distortion (Fig. IF) can also be estimated by setting 

25 a greater number of model parameters and distortion 
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can also be corrected. 

In the description of each preferred embodiment , 
the outline extraction device 101 extracts outlines 
according to the basic configuration of the present 
invention shown in Fig. 2A, the curved-surface 
estimation device 102 estimates a three-dimension 
model using outline distortion as a clue, and the 
distortion correction device 103 corrects the 
distortion based on the estimated three-dimension 
curved-surface model and outputs the corrected image. 
However, as shown in Fig. 14, it can also be configured 
so that a correction device 1402 can correct distortion 
using outline distortion extracted by an outline 
extraction device 1401 as a clue without estimating 
a curved-surface and can output the corrected image. 

The outline extraction device 1401 performs the 
same process as that of the outline extraction device 
101. However, the distortion correction device 1402 
is described using as an example, a case where an outline 
1501 as shown in Fig. 15 is obtained. First, the 
distortion correction device 1402 horizontally 
enlarges/reduces the outline 1501 in accordance with 
each line segment length of top outline 1502, 
center-folded outline 1503 and bottom outline 1503 of 
the extracted outline 1502 (Al) . Then, the device 1402 
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approximates the curved portions (1505-1 through 
1505-8) of each section surrounded by dotted lines as 
shown in Fig. 15 with a straight line and calculates 
both the inclination of the straight line and vertical 
5 reduction from the inclination. The device 14 02 further 
vertically enlarges/reduces the horizontally 
enlarged/reduced outlines according to the required 
reduction degree (A2) . In this way, it can also be 
configured so that distortion can be corrected by a 

10 two-dimension data process without estimating a 
three-dimension curved-surface model . 

However, the distortion correction method can 
also be executed using an information processing device 
(computer) as shown in Fig. 16. The information 

15 processing device shown in Fig. 16 comprises a CPU 
(central processing unit ) 1601, a memory 1602, an input 
device 1603, an output device 1 604 , an external storage 
device 1605, a medium driver device 1606 and a network 
connection device 1607, and these are connected by a 

20 bus 1608 with one another. 

The memory 1602 includes, for example, a ROM 
(read-only memory) , a RAM (random-access memory) , etc., 
and stores both a program and data to be used, for the 
process. The CPU 1601 performs necessary processes by 

25 using the memory 1602 and executing the program. 
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Specifically, the outline extraction device 101, 
curved-surface estimation device 102 and distortion 
correction device 103 are implemented by the program 
stored in the memory 1602, 

Image data, such as a slip, etc., are inputted 
to the information processing device via the input 
device 1603, such as an OHR, etc. the output device 
1604 includes, for example, a display, a printer, etc., 
and is used to out the process result, etc. 

The external storage device 1605 includes, for 
example, a magnetic disk, an optical disk, a 
magneto-optical disk, etc. The information processing 
device can store both the program and data in this 
external storage device 1605 and can use themby loading 
them into the memory 1602 as requested. 

The medium driver device 1606 drives a portable 
storage medium 160 9 and accesses the recorded content . 
For the portable storage medium 1609, an arbitrary 
computer-readable storage medium, such as a memory card, 
a floppy disk, aCD-ROM ( compact disk read-only memory ) , 
an optical disk, magneto-optical disk, etc., are used. 
Both the program and data can be stored in this portable 
storage medium 1609, and can be used by loading them 
into the memory 1602 as requested. 

The network connection device 1607 communicates 
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with anouutside device via an arbitrary network (line) , 
such as a LAW (local area network), etc., and 
transmits/receives data accompanying the 
communications. The information processing device can 
5 receive both the program and data from the outside 
device via the network connection device 1607, and can 
10 use them by loading them into the memory 1602. 
Although a single information processing device is used 
in Fig. 16, this can also be implemented by a process 

10 device consisting of a plurality of computers or a 
plurality of process devices via a network. 

Fig* 17 shows how to provide a software program, 
etc., to be executed by the information processing 
device of the present invention. The program, etc., 

15 canbe provided, for example, any of the following three 
methods (a) through (c) . 

(a) The program, etc., is installed in an 
informationprocessing device 1701, such as a computer, 
etc., and provided. In this case, the program, etc., 

20 is installed, for example, prior to shipment. 

(b) The program, etc., is stored in the portable 
storage medium 1609 and provided. In this case, the 
program, etc., stored in the portable storage medium 
16609 is stalled in the external storage device 1605 

25 of the information processing device 1701, such as a 
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computer, etc. 

(c ) The program, etc., is provided from a server 
1703 in the network 1702. In this case, basically the 
inf ormationprocessing device 1701, such as a computer, 
5 etc., downloads the program, etc., stored in the server 
1703 and obtains the program, etc. 

In this case, the server 1703 generates a 
propagation signal for propagating both the program 
and data, and transmits the signal to the information 

10 processing device 1701 via an arbitrary transmission 
medium in the network 1702. 

As described above, according to the present 
invention, the three-dimension model of a slip with 
three-dimension distortion can be estimated using both 

15 two-dimension outlines obtained from a piece of image 
(OHR image) photographed from the top and a restriction 
that the paper is rectangular, and the three-dimension 
distortion in the image can be corrected using the 
obtained three-dimension curved-surface model. In 

20 this way, since an input image can be corrected using 
the distortion of two-dimension outline obtained by 
a piece of a paper, one camera is sufficient and both 
implement cost and installation space can be reduced 
compared with the conventional technology. 
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